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Abstract — Heterogeneous face recognition (HFR) involves 
matching two face images from alternate imaging modalities, 
such as an infrared image to a photograph or a sketch to a 
photograph. Accurate HFR systems are of great value in various 
applications (e.g., forensics and surveillance), where the gallery 
databases are populated with photographs (e.g., mug shot or 
passport photographs) but the probe images are often limited to 
some alternate modality. The prototype subjects (i.e., the 
training set) have an image in each modality (probe and gallery), 
and the similarity of an image is measured against the prototype 
images from the corresponding modality. The accuracy of this 
nonlinear prototype representation is improved by projecting 
the features into a nonlinear principal component subspace. 
Four different heterogeneous scenarios are: 1) near infrared 
(NIR) to photograph, 2) thermal to photograph, 3) viewed 
sketch to photograph, and 4) forensic sketch to photograph. 

Index Terms — Heterogeneous face recognition, kernel 
principal component analysis, infrared image, thermal image, 
forensic image, viewed sketch image 


I. INTRODUCTION 

Face recognition is the process of 
recognizing/verifying a face from the given image. Detect a 
face in the image, Recognize/verify the face using database 
are the two steps used for face recognition. There are scores of 
algorithms that can be applied on an image to achieve these 
two steps. But each has its own down side. For one, many of 
these algorithms are computationally expensive and take a lot 
of time. The human face poses more problems than any 
normal object. This is primarily because; the human face 
comes in many forms, textures, features and colors. If we 
design an algorithm to detect a face, it has to be generic. It 
essentially needs to be one which is not constrained by the 
features of a human face. 

The motivation behind heterogeneous face recognition is that 
circumstances exist in which only a particular modality of a 
face image is available for querying a large database of mug 
shots (visible band face images). For example, when a 
subject’s face can only be acquired in nighttime 
environments, the use of infrared imaging may be the only 
modality for acquiring a useful face image of the subject. 
Another example is situations in which no imaging system 
was available to capture the face image of a suspect during a 
criminal act. In this case a forensic sketch, drawn by a police 
artist based on a verbal description provided by a witness or 
the victim, is likely to be the only available source of a face 
image. The main purpose of this paper is to recognize face 
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images by using kernel principal component analysis 
algorithm when heterogeneous faces like infrared images, 
thermal images, viewed sketch images, forensic sketch 
images are given as an input image. 


II. Procedure for HETEROGENEOUS FACE 
RECOGNITION 

A. Preliminary Process of an Image (or) Image 
Preprocessing 

Preprocessing images commonly involves removing 
low-frequency background noise, normalizing the intensity of 
individual particle images, removing reflections, and masking 
portions of images. Image processing is the technique of 
enhancing data images prior to computational processing. 
Preprocessing is done by converting RGB true color images 
to grayscale images by eliminating hue and saturation 
information while retaining the luminance. Grayscale images 
are used to reduce undesired distortions and performance for 
the next process like, segmentation, feature extraction. 

B. Canny Edge Detection Algorithm is used for Edge 
Detection 

Canny edge detector have advanced algorithm. It is an 
optimal edge detection technique as provide good detection, 
clear response and good localization. It is widely used in 
current image processing techniques with further 
improvements. 

Flow chart of canny edge detection algorithm 



Canny Edge Detection Algorithm 

1 ) SMOOTHING: Noise Reduction by Smoothing 

Noise contained in image is smoothed by convolving the 
input image I (i, j) with Gaussian filter mathematically, 
the smooth resultant image is given by 
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For a pixel M (i, j) having gradient magnitude G following 
(1) conditions exists to detect pixel as edge: 


Prewitt operators are simpler to operator as compared to 
sobel operator but more sensitive to noise in comparison 
with sobel operator. 

2) COMPUTE GRADIENTS: Finding gradients 


In this step we detect the edges where the change in grayscale 
intensity is maximum. Required areas are determined with the 
help of gradient of images. Sobel operator is used to 
determine the gradient at each pixel of smoothened image. 
Sobel operators in i and j directions are given as 
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These sobel masks are convolved with smoothed image and 
giving gradients in i and j directions. 

G t = Di * F(i,j) (2) 

Gj = Dj * F (:',/) (3) 

Therefore edge strength or magnitude of gradient of a pixel is 
given by 



The direction of gradient is given by 

0=ton_1 C^) < 5) 

and are the gradients in the i- and j -directions 
respectively. 

3) Non maximum suppressions: to preserve thin edges 

Non maximum suppression is carried out to preserves all 
local maxima in the gradient image, and deleting everything 
else this results in thin edges. 

For a pixel M (i , j): Firstly round the gradient direction 
nearest 45°, then compare the gradient magnitude of the pixels 
in positive and negative gradient directions i.e. If gradient 
direction is east then compare with gradient of the pixels in 
east and west directions say E (i , j) and W (i, j) respectively. 
If the edge strength of pixel M (i, j) is largest than that of E (i, 
j) and W (i, j), then preserve the value of gradient and mark 
M (i, j) as edge pixel, if not then suppress or remove. 

4) Hysteresis Thresholding: for final edges 

The output of non-maxima suppression still contains the 
local maxima created by noise. Instead choosing a single 
threshold, for avoiding the problem of streaking two 
thresholds f and f, . , are used. 

L low Is high 


If G < t low ^en discard the edge. 

If G > than thigh keep the edge. 

If t low: G < any of its neighbors in a 3 x3 region 

around it have gradient magnitudes greater than , keep fy^ h 

edge. If none of pixel (x, y)’s neighbors have high gradient 
magnitudes but at least one fall between and, f loy search f hi h 
the 5 x 5 region to see if any of these pixels have a magnitude 
greater than . If sofyeffip the edge. 

Else, discard the edge. 

C. MODULARISATION 

Modularization means dividing the image into small number 
of modules. In this method, local variations in face images are 
recognized. But, there is a conflict that whether dividing the 
image into small modules or large modules. If the image is 
divided in two very smaller modules, a large amount of sub 
region information is lost because of dependencies among 
various neighboring pixels are ignored .this problem can be 
overcome by increasing module size i.e. making the modules 
larger. But, due to this modularization the local variations in 
face images are not perfectly recognized. So, in order to 
overcome these problems, a new modularization technique 
called neighborhood defined modularization technique is 
implemented. 

1 ) NEIGHBORHOOD MODULARIZATION 

Dividing the images into smaller or larger modules create 
some problems in recognition. Here, advantages of both 
modularizations are implemented. First, Divide the image into 
large modules called sub regions. Then, divide the each sub 
region into small number of modules. Then, by merging 
neighborhood small Modules in a sub region, large numbers 
of modules are created in a sub region. So, by using this 
region based modularization feature extraction approach local 
facial Variations caused by expression variations in image can 
be dealt more effectively. 

In general, face images are captured by webcam and the face 
images are of size 1024 xl024. So, in order to process this 
image it takes a lot of time. So, to minimize the processing 
speed, the face images are normalized to a size of 64x64 in 
our experiments. Normalizing the image in terms of even 
powers of 2 also helps in dividing an image into modules of 
same size in a modular approach for face recognition. 
Modular PCA technique divides the image into non 
overlapping sub images. Then PCA is applied to each module. 
Several experimental results have been observed with 
different modules of size of 4x4, 8x8 and 16x16 on face 
images of size 64 x64. 

Finally, it is observed that, recognition accuracy is maximum 
when the image is divided into sub-regions of size 8x 8. With 
Large module size (16x16), the local variations of image are 
poorly recognized. In case of small module size (4x4), the 
dependencies among neighboring pixels are ignored. So, the 
sub-region information content is lost. Finally, it is concluded 
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that sub-region information must be considered because it 
occupies some facial feature information. 

Hence, in the implemented neighborhood defined 
modularization approach, several 8x8 modules are created by 
combining neighborhood modules of size 4x8 modules in a 
large region of size 16 xl6 in a 64 x64 face image. 

Steps followed for the implementation of Neighborhood 
defined modularization technique for an image of size MxM 
dimensions are given below: 

1 . Divide each image into non overlapping large modules of 
(M/m) x (M/m) size to get (mxm) number of large modules. 

2. Divide each large module of size (M/m) x (M/m ) into 
modules of size(M/(mxi))x(M/(mxj)), where (ixj) is the 
number of small modules within a neighborhood. 

3. Then, by merging (ixj) number of small modules in 
neighborhood, a total of R modules are created according to 
the relation^ (ixj) 1/ (k ! (ixj-k)) } where k is the number of 
small modules to be merged. 

D. FACE RECOGNITION USING KERNEL PRINCIPAL 
COMPONENT ANALYSIS 

Initially, PCA technique is used in order to extract features of 
an image. PCA is a linear subspace approach, which cannot 
capture the relation among more than two variables. Because 
PCA is directly applied to image. So PCA cannot capture the 
variations caused by expressions and other variables. In order 
to deal with this problem nonlinear space method is used. The 
non linear relationship among pixels is captured using kernel 
PCA by projecting the data into higher dimensional spaces. 


K'=K- y K.j.f - 1 K.j.f+ L 

( 12 ) 

3. Eigen values, eigen vectors computed for kernel centre 

Matrix r n 

[\V,A] = eigK' (13) 

4. Normalize the Eigen vectors 
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(14) 

5. Multiply normalized Eigen vectors with kernel centre 
matrix to get weight 
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(15) 

Apply KPCA to each module and calculate the weights for 
each individual module using Eigen vectors and kernel 
matrix. By using a minimum distance classifier classify each 
module based on the generated weights from the training and 
the testing phase for recognizing a face. 


II. SIMULATION RESULTS 


A. Input Image 

Take four different modalities like figures a. infrared image, 
b. thermal image, c. forensic image, d. viewed sketch image as 
input for matching. 


KERNEL PCA 

Input: Data X = {X lr X 2 , inn-dimensional 

space. 

Process: Mean of the data is given by 

™ = ( 6 ) 

But mean not only enough to extract features. So, covariance 
matrix C is given by 

C = Ij =1 XjX^ (7) 

The Eigen vectors corresponding to covariance matrix is 
given by 

CV = XV (8) 

Here data is projected in to higher dimensional spaces 

(9) 

Then Eigen values and Eigen vectors of covariance matrix is 

C$V 0 = X$V 0 ( 10 ) 


So feature extraction can be done by finding Eigen values and 
Eigen vectors and weights to each module as follows 
1 . Kernel matrix 


i,j = (1,2 0 


( 11 ) 
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2. Then, find kernel centered matrix .here i , j are unity 
matrices of module size 



Fig .a 

Infrared image 


Fig. b 

Thermal image 



Fig. c 

Forensic sketch image 



Fig. d 

Viewed sketch image 


B. Edge Detection four above four images 



Fig. 

Infrared image edge 
detection 


Fig. 

Thermal image 
edge detection 
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person tracked image 


person tracked image 

H 


■ 




Fig. 

Fig. 


Forensic image Viewed sketch image 

edge detection edge detection 


After preprocessing commonly involves removing 
low-frequency background noise, normalizing the intensity of 
individual particle images, removing reflections, and masking 
portions of images. Image processing is the technique of 
enhancing data images prior to computational processing an 
image. Canny edge detector algorithm is used for edge 
detection to good detection, clear response and good 
localization as shown in the above figures for four modalities. 

C. Face Recognition for above four images 

After modularization, in pattern recognition feature 
extraction is the most difficult problem; its reason is that we 
want to obtain the best features with minimum classification 
error and low running time. Initially, PC A technique is used in 
order to extract features of an image. PC A is a linear subspace 
approach, which cannot capture the relation among more than 
two variables. Because PCA is directly applied to image. So 
PC A cannot capture the variations caused by expressions and 
other variables. In order to deal with this problem nonlinear 
space method is used. The non linear relationship among 
pixels is captured using kernel PCA by projecting the data 
into higher dimensional spaces. So apply kernel PCA to each 
module to extract features of each module and finally 
calculate the weights of each module by using kernel matrix 
and algorithm. By using a minimum distance classifier 
classify each module based on the generated weights from the 
training and the testing phase for recognizing a face. The 
image which is matched based on generated weights gives us 
an output of image matched else it gives face not in database. 



III. Conclusion 

In this paper four different heterogeneous scenarios like 
infrared images, thermal images, viewed sketch images, 
forensic sketch images are given as an input image. First 
preprocessing is done for given image to reduce undesired 
distortions and enhance the data for further process .Then 
sobel operator is used for edge detection to make the 
recognition system independent to the illumination. Apply 
modularization to get local variations in an image. So feature 
extraction can be done by finding Eigen values and Eigen 
vectors and weights to each module by applying kernel 
principal component analysis ( kpca) method. To recognize a 
face minimum distance classifier is used to classify each 
module based on the generated weights from the training and 
the testing phase. Therefore by using kernel pea algorithm for 
recognition rate is improved. In particular, the KPCA method 
will be useful for identification Systems subjected to large 
variations in illumination and facial expressions and also for 
different modalities. 
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Fig. c Fig. d 

The above fig. a, fig. b, fig. c, fig. d are corresponding gallery 
photographs i.e. visible band face image, called VIS of above 
given input images. 
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